%CHAPTER 4
\chapter{Distributed simulation with HLA}
\label{chap:DistributedSimulation}
The simulations of large systems often face with the performance issues. The use of Cuda programming model can deal with those. However, the lack of interoperability between simulations poses a major challenge. Thus, the High Level Architecture [(HLA)~\cite{IEEEHLARules},~\cite{IEEEHLASpecification},~\cite{HLAAnIntroduction}] standard is proposed as a solution for addressing that new demand. According to this standard, the distribution of many sub-simulations can be achieved instead of the development of one vast simulation. The integration of Cuda model and the HLA leads to a hybrid solution in which several parallel simulations can be distributed on different computer systems. This chapter gives a brief description of the application of HLA on parallel simulations.
\section{Overview of The High Level Architecture (HLA)}
\label{highlevelarchitecture}
The High Level Architecture (HLA) (~\cite{IEEEHLARules},~\cite{IEEEHLASpecification},~\cite{HLAAnIntroduction}) is a standard for distributed simulations, the main goal is to support interoperability and reusability of simulations. The HLA was developed by the United States Department Defense (DoD) to facilitate the integration of distributed simulation models within an HLA environment. It allows the division of a large scale model into a number of manageable components, while maintaining interaction between them. Over the last years, the HLA is deployed in a wide range of simulation application areas including transportation and the manufacturing industry. But, it hardly appears in simulation about phenomena, especially the climate change area. The HLA is thus suggested as a potential approach of composition of parallel simulations in this project.
\begin{figure}[H]
	\begin{center}
		 \includegraphics[width=12cm]{img/hlastructure.png}
		 \caption{HLA Federation.}
		 \label{img:hlastructure}
	\end{center}
\end{figure}
In HLA terminology, the entire system is represented by a \textit{federation}. Each simulator referring to the \textit{federation} is called a \textit{federate}. A set of \textit{federates} is connected via \textit{Run Time Infrastructure} (RTI). These \textit{federates} can be established on different platforms and connected together by a network system. In such case, RTI can be viewed as distributed operating systems for interconnect cooperating system \textit{federates}. Figure~\ref{img:hlastructure} describes the global architecture of a HLA simulation. Generally, the HLA specification defines: 
\begin{itemize}	   
	   \item\textbf{A set of rules:} This describes the responsibilities of federates and their relationship with RTI. There are ten rules. One of them is that \textit{all exchange of data among federates should occur via the RTI during a federation execution}. 
       \item\textbf{An interface specification:} The interface specification prescribes the interface between each federate and the Runtime Infrastructure (RTI), which provides communication services to the federates. The interface specification is divided into some main management areas: 
       \begin{itemize}
       		\item\textit{Federation management:} Federation management includes main tasks such as creating federations, joining federates to federations, resigning federates from federations, and destroying federations.
       		\item \textit{Declaration management}: This allows federates publish and subscribe class attributes and interactions to RTI. Other federates can only subscribe to an attribute or an interaction when they were published by the federates owning them.
       		\item\textit{Object management:} Which includes the tasks of creating, and sending the updates of objects to other federates.
       		\item\textit{Ownership management:} The RTI allows federates to distribute the responsibility for updating and deleting object instances with a few restrictions. 
       		\item\textit{Time management:} This focuses on the implementation of time management policies and negotiate time advances. This mechanism allows to create several simulations running concurrently.       
       \end{itemize}
       \item\textbf{An Object Model Template (based on the OMT standard~\cite{IEEEHLAOMT}):} This component defines how information is communicated between federates, and how the federates and federation have to be documented (using Federation Object Model FOM). FOM defines the shared objects, attributes, and interactions for whole federation. 
\end{itemize}
There are two elements can be exchanged between federates:
\begin{itemize}
	\item[] \textbf{An object:} is an entity that represents “\textit{actor}” playing in the simulation. It contains shared data that are created by a federate during the federation execution and persist until it is destroyed. The FOM defines all classes of object, a case presented in Table~\ref{table:PublishAndSubcribe}. As a federate wants to publish or subscribe to an object, it must compatibly define that object in its FOM. Objects store their data in attributes.
	\item[]\textbf{An interaction:} is a broadcast message that any federate can send or receive. A publishing federate sends out an interaction to the federates, which have subscribed to the publisher. If no subscribing federate receives the interaction, the data it carries are lost. The FOM also defines all classes of interaction. As a federate wants to publish or subscribe to an interaction, it must compatibly define that interaction in its FOM. Interactions carry data in parameters.
\end{itemize}
\begin{figure}[H]
			\begin{center}
		 		\includegraphics[width=9cm]{img/federate_federation.png}
		 		\caption{Illustrating a high level of the interplay between a federate and a federation.}
		 		\label{img:federate_federation}
			\end{center}
\end{figure}
Figure~\ref{img:federate_federation} depicts the interplay between a federate and a federation. Initially, a federate will try to create a federation, or to connect to existing one on RTI. It then specifies what data will be shared with other federates by using publishing services. These published objects or published interactions will be available to all federates, which also has a connection to the same federation.\\
An federate want to send data to other federates, it has to register objects and call an update service. That data will be automatically reflected to subscribers by the RTI. Releasing allocated resources is always necessary at the end.
%----------------------------------------TIME MANAGEMENT-------------------------------------------
\section{Time management in HLA}
\label{section:TimeManagement}
The RTI provides a variety of optional time management services. It is important to understand time management to manage the mechanism of exchanging events between federates. Each federate manages its own logical time and communicate this time to the RTI. The RTI will ensure correct coordination of federates by advancing time coherently. In the discrete event simulation literature, logical time is equivalent to \textit{"simulation time"}. It is used to make sure that federates observe events in the same order~\cite{HLATimeManagement}. It helps to avoid many problems such as causality violation, or different results led from repeated executions of the simulation with the same input data. Logical time is not mapped to real time.
\subsection{Time policies} 
According to the HLA time policies, each federate is involved in the progress of time. In some cases, it is necessary to map the progress of one federate to the progress of another. A federate needs to request a \textit{regulation} policy to participate in the decision for the progress of time. A \textit{constrained} federate follows the time progress imposed by other federates. As our approach, the synchronization of logical time from different federates is necessary. Thus, the \textit{federating} and \textit{constrained} federates are allowed, as shown in Table~\ref{img:TimeManagement}. This enables participating federates can exchange data together.
\begin{comment}
To deal with the time progress and ensure causality of several federates, Time-Stamp Ordered (TSO) events are proposed by HLA. A TSO event is simply an event with an associated timestamp. Only a \textit{regulating} federate is able to generate TSO events. It makes sure that any TSO events it generates will occur no earlier than the current time plus the \textit{lookahead}, "t\textsubscript{current}+t\textsubscript{lookahead}".\\% The \textit{lookahead} value is considered as a contract between the \textit{regulating} federate and the federation.\\
In HLA, the RTI is responsible for the ordering of the TSO events, which are generated by federates of the federation. TSO events stored into a queue within the RTI, but are not eligible for delivery until there are no other TSO events intended for that federate having a smaller timestamp.
\end{comment}
\subsection{Time progress}
The second portion of the time management component provides a mechanism to advance simulation time within each federate.
There are two particular services which federates can invoke to request time advancement from the RTI. The \textit{timeAdvanceRequest} is used to implement time-stepped federates; the \textit{nextEventRequest} is used to implement event-based federates. The granted time is given by \textit{timeAdvanceGrant} service.\\
Generally, a time management cycle consists of three steps. First, a federate sends a request for time advancement. Next, the federates can receive \textit{ReflectAttributeValues} callbacks. The RTI completes the cycle by invoking a federate defined procedure called \textit{timeAdvanceGrant} to indicate the federate’s logical time has been advanced.
\begin{figure}[H]
	\begin{center}
		 \includegraphics[width=7cm]{img/TimeAdvanceRequest.png}
		 \caption{A model of time advancement request is used in this project.}
		 \label{img:TimeManagement}
	\end{center}
\end{figure}
\subsection{Time synchronization}
As presented in previous sections, all simulations have to synchronize their local logical time to ensure the causality. The \textit{constrained} parameter and \textit{regulating} parameter are enabled for all simulations. The former ensures federates to be able to send the updates and interactions in causal order. In the other hand, the latter allows federates to able receive those updates and interactions from the RTI. Since a passive visualization federate does not send any updates or interactions, it has no impact to the time advance of the federation. Therefore, only \textit{constrained} parameter is enabled and \textit{regulating} can be switched off in the case of visualization. Table~\ref{table:TimeManagement} shows time policies proposed for the case study (Section ~\ref{section:CaseStudy}).
\begin{table}[H]
\begin{center}
\begin{tabular}{|l|l|l|l|}
\hline 
\textbf{Federate} & \textbf{Time constrained} & \textbf{Time regulating} & \textbf{Time advance} \\ 
\hline 
Forest & Yes & Yes & Time stepped \\ 
\hline 
River & Yes & Yes & Time stepped \\ 
\hline 
WSN & Yes & Yes & Time stepped \\ 
\hline 
Visualization & Yes & Yes/No & Time stepped \\ 
\hline 
\end{tabular} 
\caption{Time management of the federation.}
\label{table:TimeManagement} 
\end{center}
\end{table}
To synchronize activities between several federates participating in a federation, the RTI gives a mechanisms for exchanging data between them. In this case, times will be associated with exchanged data in coordinating federate activities. The RTI allows federates communicate explicit synchronization points. Figure~\ref{img:Synchronization} illustrates a process of synchronizing between two federates, the river federate and the forest federate. 
\begin{figure}[H]
	\begin{center}
		 \includegraphics[height=6cm]{img/Synchronization.png}
		 \caption{Federate synchronization}
		 \label{img:Synchronization}
	\end{center}
\end{figure}
First of all, one of available federates sends a synchronizing request to the RTI and in this case it is the river federate. Then, the RTI will send the response to river federate and later send an announce to other federates to achieve a synchronization point. A service will be used by federates to confirm the synchronized point achieved.
\begin{comment}
As illustrated in Figure~\ref{img:TimeManagement}, FederateTAR produces an event at time \textit{t\textsubscript{1}} by invoking service \textit{updateAttributeValue (UAV)}. The current logical time is \textit{t\textsubscript{1}}. We assume that the next event is \textit{e\textsubscript{2}} with timestamps \textit{t\textsubscript{2}}, \textit{t\textsubscript{2}} = \textit{t\textsubscript{1}} + $\Delta$ ($\Delta > 0$). The federate sends a request \textit{TAR(\textit{t\textsubscript{2}})} to the RTI to advance local time. After that, it can receive a \textit{RAV(v,t\textsubscript{1}')} callback. Timestamp \textit{t\textsubscript{1}'} is in the interval [\textit{t\textsubscript{1}},\textit{t\textsubscript{2}}]. The federate will advance to \textit{t\textsubscript{2}} until the \textit{TAG(t\textsubscript{2})} is returned. The federate then can increase its local time.\\
FederateNER also produces an event \textit{UAV} at time \textit{t\textsubscript{1}} and sends a request to advance to \textit{t\textsubscript{2}} by \textit{NER(t\textsubscript{2})}. The reception of \textit{RAV(v,t\textsubscript{1}')} is followed by a \textit{TAG(t\textsubscript{1}')}. The federate advances its local time to \textit{t\textsubscript{1}'} and can perform a computation with received value. It is noted that the federate using \textit{NER} service is more reactive because it can produce new events from time \textit{t\textsubscript{1}'}.
\begin{figure}[H]
	\begin{center}
		 \includegraphics[width=9cm]{img/TimeManagement.png}
		 \caption{Time advancement services a.TAR b.NER.}
		 \label{img:TimeManagement}
	\end{center}
\end{figure}
\end{comment}
In the next portion, some issues relating to exchanging data is considered in the context of distributed simulations. 
\subsection{Exchanging data}
\label{section:exchangingdata}
Exchanging data between simulation federates is one important part of distributed systems. However, a question arriving in this case is that what kind of data must be shared, where the communication will happen.\\
Regarding the type of exchanging data, it is determined by characteristics of real systems as well as interoperability between them. As our case, there is a communication between four federates: \textbf{forest, river, WSN, } and \textbf{visualization}. The forest federate transports its status to the river federate and the WSN federate. Meanwhile, the three federates need to provide their data to the visualization federate for analyzing the results.\\
To achieve it, the forest federate will publish its data (forest status and position) as an object class (\textbf{ForestNode}). The river federate and the WSN federate need to subscribe it. As the same case, the river federate and the WSN federate also publish the object classes \textbf{RiverNode} and \textbf{WSNNode}, respectively. These published classes have to be declared in the FOM of the federation, has a structure as indicated in Table~\ref{table:PublishAndSubcribe}.
\begin{table}[H]
\begin{center}
\begin{tabular}{|l|l|l|l|}
\hline 
\textbf{Object Class} & \textbf{Attributes} & \textbf{Published by} & \textbf{Subscribed by} \\ 
\hline 
ForestNode & State, Position & ForestNode & River, WSN, Visualization \\ 
\hline 
RiverNode & Pollution density, Position & RiverNode & Visualization \\ 
\hline 
WSNNode & State, Position & WSNNode & Visualization \\ 
\hline
\end{tabular} 
\caption{Objects and their attributes, publishers and subscribers.}
\label{table:PublishAndSubcribe} 
\end{center}
\end{table}
In some cases, it is also important to specify where data will be exchanged between two federates, especially in the case of physical systems owning very large sizes. Indeed, it is often inefficiency to send the entire data via the RTI because of issues with network performance and local computation yield. Thus, we proposed a solution for a general case of exchanging data between two adjacent systems.\\
Adjacent situation is two physical systems that have a common frontier or some places in common. It is useless if unrelated information is sent to others. For example, as shown in Figure~\ref{img:UMinhThuong}, new polluted points to the river can be caused by the ashes of forest fire only appears at the frontier of the two systems. Forest fire federate sends regularly its states to the river federate. The latter only takes care states of points close to it instead of entire forest states. This not only takes time for transporting data between federates, but also lead to less efficient in computation at receiver side.\\
A solution based on the morphology theory~\cite{PhormologyTheory} can be used to address that issue. That enables to smooth the boundary of physical systems by applying basic operations such as \textit{erosion}, and \textit{dilation}. To summarize, only the status data at the boundary of forest will be sent to the RTI.
%However, this solution is only suggested as the communications of physical systems appear at frontiers.
\section{Distributed physical simulation}
This section presents an application of using of the HLA standard for unifying parallel several simulations, or called a mixed simulation. The study region was suggested as shown in Figure~\ref{img:UMinhThuong}.\\
The whole model was split in three simulation federates: \textbf{forest fire spread, river pollution diffusion}, and \textbf{WSN}. The simulation federates was all implemented in accordance with the Cuda programming model and the HLA standard as well. These parallel simulations are executed concurrently as three different simulators. Their models were presented in Section~\ref{section.SimulationSystem}.\\
In addition to the three federates, the last one, \textit{visualization}, is designed as a supportive federate. The overview about the federation can be seen in Figure~\ref{img:overviewsystem}.
\begin{figure}[H]
	\begin{center}
		 \includegraphics[width=12cm]{img/overviewsystem.png}
		 \caption{A structure for a proposed federation.}
		 \label{img:overviewsystem}
	\end{center}
\end{figure}
Repeating the communication that was proposed in Section~\ref{section:CaseStudy}, \textbf{forest fire spread} will produce ashes which result in some new polluted points and dusts to \textbf{river pollution diffusion} at time t. The latter will include these new data to its model at time t+1. The communication depends on a specify condition.\\
There is also the communication between \textbf{WSN} and \textbf{forest fire spreading}, the sensors regularly collect the forest status as it is the goal of sensing. New information will be sent to observers. In the case of fire detected, the observers will raise emergency signals as the fires were detected. To do that, the synchronization needs to be achieved as indicated in Table~\ref{table:TimeManagement} and the shared data have to be declared as shown in Table~\ref{table:PublishAndSubcribe}.\\
The file FOM for the federation was represented in the \textbf{cyber.fed} file shown in~\ref{lst:fedfile}. 
\begin{lstlisting}[caption=cyber.fed file, label=lst:fedfile]
;; Cyber physical simulation
(Fed
  (Federation Cyber)
  (FedVersion v1.0)
  (Federate "river" "Public")
  (Federate "forest" "Public")
  (Federate "wsn" "Public")
  (Federate "visualization" "Public")
  (Objects
    (Class ObjectRoot
      (Attribute privilegeToDelete reliable timestamp)
      (Class RTIprivate)
	(Class ForestNode
		(Attribute PositionX RELIABLE TIMESTAMP)
		(Attribute PositionY RELIABLE TIMESTAMP)
		(Attribute State RELIABLE TIMESTAMP)
	)
	(Class RiverNode
		(Attribute PositionX RELIABLE TIMESTAMP)
		(Attribute PositionY RELIABLE TIMESTAMP)
		(Attribute Density RELIABLE TIMESTAMP)
	)
	(Class SensorNode
		(Attribute PositionX RELIABLE TIMESTAMP)
		(Attribute PositionY RELIABLE TIMESTAMP)
		(Attribute State RELIABLE TIMESTAMP)
	)
    )
  )
)
\end{lstlisting}

\subsection{Forest fire spread federate}
The model of this simulation federate was presented in Chapter~\ref{chap:PhysicalSimulation}. In which, there are some fires (red points) being randomly initialized in the forest. These fires will spread around according to the transition function and CA pattern of the model. An example about the spreading is shown in Figure~\ref{img:PollutionDiffusion}. The green, red, grey, and white points represent the \textit{trees, fires, ashes}, and \textit{empty} states, respectively.
\begin{figure}[H]
	\begin{center}
		 \includegraphics[height=7cm]{img/ForestFederate.png}
		 \caption{An example of simulating of fire spread in the forest. The pattern of 4 neighbour is used. The red color represents fire trees and the gray color implies ashes formed by the fire.}
		 \label{img:PollutionDiffusion}
	\end{center}
\end{figure}
The \textit{ashes} can be formed after some steps. These \textit{ashes} are able to pollute the river as shown in Figure~\ref{img:Communication_Forest_River}. 
\subsection{River pollution diffusion federate}
The model of pollution diffusion in the river was also presented in Chapter~\ref{chap:PhysicalSimulation}. Initially, there are some polluted points randomly generated in the river. During the progress of diffusion, it always receives status data about the fire from forest federate via the RTI. It will check the data to determine whether the ashes will pollute some river cells or not. This defends on a specific condition. For each river cell, if the distance to an ash cell is equal or less than a specify threshold, the pollution density of river cell will decrease in inverse proportion of that of the distance.\\
The RTI only sends that data to river federate as soon as it receives an update call from forest federate. The update call only appears when ashes presented in the scope of the forest boundary.
\begin{figure}[H]
	\begin{center}
		 \includegraphics[height=9cm]{img/Communication_Forest_River.png}
		 \caption{A result is got from visualization federate. This demonstrates the exchanging data between the two simulations via the RTI. Two regions marked with the red circles representing the new pollution created by the \textit{ashes}, which are formed from the forest fire after 4 steps.}
		 \label{img:Communication_Forest_River}
	\end{center}
\end{figure}
\subsection{WSN federate}
The model of WSN was also introduced in Chapter~\ref{chap:PhysicalSimulation}. Every time step, nodes will receive the data from forest federate via the RTI and only consider to cells in the scope of the sensing range. If it detected that there are fire, it will forward that information to a observer for making decision. The signals will be raised as the fire is recognized. As depicted in Figure~\ref{img:WindowFederate}, the red rings indicate that fires have been detected at those sensors.
\subsection{Visualization federate}
The viewer federate is based on the 2D visualization X Window System. As mentioned above, it first subscribes all necessary data, which have been published by other federates. The aim is to provide a overview on the results as shown in Figure~\ref{img:WindowFederate}. Initially, the background of the viewer is drawn from visible data extracted from PickCell tool. During the federation execution, this federate will receive data from others and update the view at every step. 
\subsection{A case study}
This section describes a case of the federation. Initially, one federate creates a federation on the RTI and waits for other federates to participate. Another federate will connect to that federation and also wait until the last coming. The first one will send a request to others to achieve a synchronization point. After the responses of other federates, the synchronization point is achieved. They run on the same time progress. At each time step, these federates exchange data together via the RTI. Figure~\ref{img:WindowFederate} presents the results captured from visualization federate.
\begin{figure}[H]
	\begin{center}
		 \includegraphics[height=10cm]{img/WindowFederate.png}
		 \caption{Illustrating an interoperability between the four federates via the RTI.}
		 \label{img:WindowFederate}
	\end{center}
\end{figure}
\subsection{Simulation tools}
Along with the PickCell tool, which is developed at LabSTICC laboratory. An Open Source software, CERTI~\cite{certi}, was used in this project. The CERTI RTI supports HLA 1.3 specification (C++ and Java). The X Window System was used to support for displaying the results of simulation federates.